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The index hypergeometric transform (it is also called the Olevsky transform and 
the Jacobi transform) generalizes the spherical transforms in on rank 1 symmetric 
spaces (i.e. real, complex and quaternionic Lobachevsky spaces). The purpose of the 
present work is to obtain properties of the index hypergeometric transform imitating 
the analysis of Berezin kernels on rank 1 symmetric spaces. 

We also discuss a problem of explicit construction of a unitary operator identifying 
the space and a Berezin space. The problem is reduced to some integral expres- 
sion (the A-function) that apparently can't be expressed in a finite form in terms of 
standard special functions (only for some special values of parameters this expression 
can be reduced to the so-called Volterra type special functions). We investigate some 
properties of this expression. We show that for some series of symmetric spaces of large 
rank the operator of a unitary equivalence can be expressed through the determinant 
of some matrix consisting of the A-functions. 



0.1. Mehler Weyl Titchmarsh Olevsky index hypergeometric trans- 
form. Fix 6, c > 0. Consider the integral transform J^^c defined dy the formula 



g{s) = Jb^cfis) 
1 



T{b + c) 



f{x) 2Fi{b + is,b- is; b + c; -x)x''+''-' {1 + xf-^dx, (0.1) 



where 2Fi{-) is the Gauss hypergeometric function. The inverse operator J^^l 
is given by 



g{s) 2Fi{b + is,b — is; b + c; —x) 



Tib + is)T{c + is) 2 



ds. 
(0.2) 



This pair of the inverse integral transforms was discovered by Weyl |Q in 
1910. Weyl's result didn't attract serious interest and again these transforms 
appeared in the works of Titchmarsh H in 1946 and M.N.Olevsky |^ 1949. 
Spherical transforms for all Riemannian noncompact symmetric spaces of rank 
1 have the form (0.1) for some special values of the parameters b.c. This was 
the main reason of the interest in the transform (0.1) in subsequent years. 

1 Supported by the grants RFBR 98-01-00303 and NWO 047-008-009 
■^The partial case b = 1/2, c = 1 (Mehler-Fock transform) was discovered by Mehler 
in 1881. 



For a detailed information on the transform (0.1) see the papers of Koorn- 
winder and Flensted- Jensen [|lOj, p^-p8[, see also the books of Yakubovich, 
Luchko p9t], Samko, Kilbas, Marichev [p3| and Tables of Prudnikov, Brychkov, 



Marichev | |32| , vol.5. Koornwinder's survey ]17| contains a large bibliography. 

For various generalizations of the transform (0.1) see, for instance, the works 
of Yakubovich, Luchko (the Wimp transform with G-function), Koelink, 
Stockman |l^ (g-analogs), Heckman, Opdam |3l] (multivariate ana- 

logues). 

In mathematical literature there exist the following terms for the transform 
(0.1) 

— the index hypergeometric transform 

— the generalized Fourier transform 

— the Olevsky transform 

— the Jacobi transform^ 

— the Fourier- Jacobi transform 

0.2. Contents. The space on a noncompact Riemannian symmetric 
space G / K admits a natural deformation discovered (in the case of Hermitian 
symmetric spaces) by Berezin §, see also 0, ||, §, |l| and a 

bibliography in . For small values of the deformation parameter (or for large 
values of our parameter 6, see §3) the Berezin representations of the group G 
are equivalent to the natural representation of G in L?{G/ K). 

The Plancherel formula for the Berezin deformation for rank 1 spaces was 
obtained by van Dijk and Hille in Q and in the general case by the author 

One of purposes of the present work is to apply these results to the theory 
of the index hypergeometric transform. 

The second purpose is to obtain an explicit construction of a unitary in- 
tertwining operator from L^{G/K) to a Berezin space for large values of the 
parameter 9. We also apply the theory of the index hypergeometric transform 
for investigation of this operator. 

Our §1 contains preliminaries on the index hypergeometric transform. 

In §2 we calculate the images of the operators 

f{x) xf{x); f{x)^x{x + l)^f{x) 

under the index hypergeometric transform. We show that these operators cor- 
respond to difference operators in the imaginary direction with respect to the 
variable s. For instance, the image of the multiplication by x is 

{b-is){c-is) {b + is){c + is) 

^'^'^ ^ (-2^,s)(l-2^.)(^^^ + ' '^'^^ + (2z.)(l + 2i.) " " '^'^^ 

Although these results are very simple, I couldn't find the statement on 
x{x + 1)^ in a literature. The statement on x is a very partial case of one 
recent Cherednik theorem 101. 



•^The term was introduced by Koornwinder by an association with expansions in Jacobi 
polynomials. 
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In §3 we give a brief survey of analysis of Berezin kernels on rank 1 symmetric 
spaces. Below we imitate this section on the level of the index hypergeometric 
transform. 

In §4 we give preliminaries on the continuous dual Hahn polynomials. 

In §5 we obtain a family of nonstandard Plancherel formulas. Precisely, 
we show that the index transform is a unitary operator from some space of 
holomorphic functions in a disk (or in a half-plane) to the space on a half- 
line with respect to some measure. This section is an imitation of the works Q , 
[ p4| on a level of special functions of one variable. 

In S6 we define the function 



1 [ r(a + is)T{b + is)V{b - is)T{c + ts)T{c ~ is) 

'''^^ ' nr{b + c)J r{2is)r{-2is) 







X 2Fi{b + is,b — is;b + c: —x) ds 



Our purpose is to construct a unitary intertwining operator from to a Berezin 
space. Apparently, the function A can't be expressed explicitly by means of 
standard special functions (with exception of some special values of 5, c, when 
A can be reduced to the rare Volterra type special functions). We also try to 
understand an answer to the following informal question: is it natural to consider 
the A-function as a "new" special function? We find a collection of identities 
with A. In §7 we construct some orthogonal bases in on the half-line by 
means of the A-function. 

In §8 we present (without a proof) an explicit construction of a unitary 
operator from to the Berezin deformation of for the symmetric spaces 

\J{p,q)/V{p)xV{q). 

It turned out that this integral operator is expressed by a determinant of a 
matrix consisting of the A-functions. This section explains the purposes of 
introduction of the A-function. 

0.3. Notation. M+ denotes the half-line x ^ 0. 



TP f L \ \ " («)n(^)n ri 

2Fi{a,b;c;x) -.^ ^ x 

n=0 y'^l^^- 



is the Gauss hypergeometric function. 

(ai)„ . . . {ap)r 



ai, . . . ttp 

]X 

C-l, ■ ■ ■ 1 Cq 



(ci)„...(cg)„n! 
is the generalized hypergeometric function. 

§1. Preliminaries 

1.1. Definition. Fix 6, c > 0. Let f{x) be a function on the half-line 
M+. Its index hypergeometric transform Jb.cf (we also use the notation [/]b,c) 
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is given by 



.g(s) = Jb^cfis) = [fis)]b,c = 

1 



fix) 2Fi{h + is,b- is; b + c; -x)x''+''-\l + xf-^dx. (1.1) 



r(6 + c) 

The inversion formula for the index hypergeometric transform is 



1 



^r(6 + c) J, 



g{s) 2Fi{b + is,b — is; b + c; —x) 



T{b + is)T{c + is) 



T{2is) 



ds. 
(1.2) 



This is equivalent to the following statement: 
Jb c is a unitary operator 



r-b + C-l, 



1 + x)''^''dx 



^+1 



T{b + is)Tic + is) 



T{2is) 

This property (the Plancherel formula) in the explicit form is 



dx 



flix)f2ix)x''+'-\l+x)'--dx - - / [fl{s)]bAf2{s)] 



T{b + is)r{c + is) 



T{2is) 



ds. 



Behavior of the operator J;,.c in different functional spaces are known in details, 
see the survey ||l^ , and also |^ . 

Remark. Weyl's ([^) proof of the inversion formula (1.2) is relatively 
simple. Consider the hypergeometric differential operator 

D := x{x + l)—+[{c + b) + {2b + l)x\ ■^+b^ 

on R+. The functions 2-Fi(^ + is, b — is;b + c; ~x) are the eigenf unctions of 
D. There exists an explicit formula for the resolution R{X) — {D — X)~^ of the 
operator D (this is equivalent to an explicit expression for the Green function, 
see Myller-Lebedeff 's works [^-[|3|). If we know the resolution, then we can 
easily find the spectral expansion by the formula 

— [ lim \R(X + ie) ~ R(X - ie)] dX 
27ri J e^o'- \ ' ^ 'J 

A detailed discussion of this reception is contained in Dunford, Schwartz Q. 
For differential operators (and in particular for the hypergeometric operator) 
this method is present in Titchmarsh's book p^ . 

For other proofs of the inversion formula see |3^, ||l^-|18 , |Q, [ p9[ . 

Remark. Obviously, 

m = s'[ns)] 
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1.2. Holomorphic extension to the strip. 
Lemma 1.1. Let f be integrahle on and 

fix) ^ o{x-°'-'), x^+oo 

where e > 0. Then [/(s)]f,.c is holomorphic in the strip 

I Im s| < a — b 

Proof. This is a consequence of the following asymptotics for the hyperge- 
ometric function (see |9|, vol. 1, (2.3.2.9)) as a; — > +oo 

2Fi{b+is,b-is;b+c;-x) = \ix-''+"' +\2X-''-"' + 0{x'''+"-^) + Oix-''-"'-^), 

where 2s ^ Z and Ai, A2 are some constants (for 2s e Z there appears the factor 
In X in the front of the leading term) . 

Remark. Assume a continuous function f{x) satisfies the condition 

f{x) = o{x-''-^), x-^+oo. 

Then f{x) is an element of L^(IR+) with respect to the measure x^~^'^~^{l + 
x)^~'^dx. We stress that this condition is also sufficient for existence of analytic 
continuation to a narrow strip. 

1.3. Fourier transform. For some special values of the parameters b, c the 
function 2^1 (^ + is, 6 — is; b + c; —x) is an elementary function. In particular 
(see §, vol 1, (2.8.11-12)), 

2Fi{is,—is; ^; —x) — cos(2s arcsh-^a;), (1.3) 

^ ,1 .1 . . sin(2s arcsh -v/a;) ,s 

2Fi(l + zs,|-^s;|;~a;) = ^ 2sy/^ ^ ^ ^ 

Hence, 

1 f°° 

JoA/2f{s) - /(a:)cos(2sarcshV^)x-i/2(l + x)-i/2^x, 

7 f^ \ ^ [°° ft Nsin(2sarcshV^) W2 ^-l/2 , 

= ^ m — — x'{i+x) I dx. 

The substitution x ~ sh^ y reduces these integrals to the form 



Jo,i/2fis) = Y[i/2) X -^^^^ y)cos{2sy)dy, 

T fl \ 1 r fl 1,2 ssliysin(2sy) 

= ^ /(Bh ,) dy, 

i.e. the operator Jo, 1/2 (resp. Ji/2,1) differs from the cos-Fourier transform 
(resp. the sin- Fourier transform) by a nonessential variation of the notation. 
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Notice that the following functions also are elementary 

2-Fi(fc + is,k — is; ^ + k + I; —x); 2Fi{^ + k + is, ^ + k — is; ^ + k + I; —x). 

for integer values of fc, I. They can be obtained from (1.3)-(1.4) by an application 
of suitable explicit differential operators (see ||^, vol. 1, 2.8, formulas (20), (22), 
(24), (27)). 

1.4. Spherical transform. Denote by K the real numbers M, the complex 
numbers C, or the quaternions H. Denote by r the dimension of the field K. By 
K" we denote the n-dimensional space over K with the standard scalar product 

Denote by U(l,n;IK) the pseudounitary group over K, i.e. the group of all 
(1 + n) X (1 + n) matrices ( " ^) over K, satisfying the condition 

fa b\ f-l 0\ fa b\* _ f-l 

\c d) yo ijyc d) ^yo i 

The standard notations for U(l, n; IK) in the cases K = M, C, H are respectively: 
0(l,n), U(l,n), Sp(l,n). 

By B„(IK) we denote the unit baU (z, z) < 1 in K". Let 5'^""^ be the unit 
sphere {z,z) = 1. The group U(l,n;K) acts on B„(]K) by the linear fractional 
transformations 

z := (a + zc)-i(& + zd). (1.5) 

The stabilizer K of the point G B„(K) consists of the matrices 

|a| = 1, de\J{n;K). (1.6) 

Hence B„(K) is a homogeneous space, 

B„(IK) = U(l,n;IK)/U(l;IK) x U(n;IK). 
The Jacobian of transformation (1.5) is 

J{g;z) = |a + zcr''(i+"). 
The following formula can easily be checked 

1 - (2[sl,u[9l) = (a + zc)~i(l - {z,u)){a + uc)-\ 
This implies that the U(l, rt; K)-invariant measure on B„(K) has the form 

dm{z) = (1- (z,z))-("+i)'-/2dz, 
where dz denotes the Lebesgue measure on B„(K). 
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The group U(l,n;K) acts in L^(B„(K), (im(^)) by the substitutions 

f{z)^ f{{a + zc)-\b + zd)). (1.7) 

Consider the problem of the decomposition of i^(B„(]K), dm{z)) into a direct 

integral of irreducible representations of the group U(l,n;K). 

Let s € M. The representation Ts of (spherical) principal unitary series of 
the group U(l,n;K) acts in //^(S™"^) by the operators 

^«(c ^d)^ih) = f{{a + hc)-\b + hd))\a + hc\-^^+'^^/^+'+'\ (1.8) 

where h e 5™-^ 

Consider the operator A from the space i^(B„(]K), rfm(2:)) to the space of 
functions on 5'''"^^ x M+ given by 

r \\ - (z h\\-in+l)r/2+l+2s 

Afih,s)= / f{z)-] ; \ >o dz. (1.9) 

If / is transformed by (1.7), then G{h, s) := Af{h, s) is transformed by 
G{h,s)^G{{a + hc)-\b + hd), s)|a + /icr("+i)"/2+i+«_ 

Now wo want to construct the measure dv (it is called the Plancherel mea- 
sure) on S'''"~^ X M+ such that the operator A is a unitary operator 

L2(B„(K),dm(^)) ^ L2(5.rn-1 X 

(this will solve the problem of decomposition into a direct integral) . Obviously, 
the factor \a + hc\ (.(1.8), (1.9)) is 1 if the matrix (^ ^) has the form (1.6). Thus 
the required measure dv is invariant with respect to the group U(n;K). Hence 
the measure du has the form 

a{s) ds dh. 

To find the function a{s), we consider all functions / on B„(K) depending only 
on the radius \z\. It is convenient to introduce the variable 



U|2 



X 



1 - \z\ 



and to assume / = f{x). Then the corresponding function G{h, s) depends only 
in the variable s and after a simple calculation we obtain 

/•OO 

G{s) = const • / fix) 2Fi{b + is,b- is; b + c; -x)x^+''-^{l + x)^-''dx, 
Jo 

(1.11) 

where 

6= (n+l)r/4-l/2; c = (n - l)r/4 + 1/2. (1.12) 
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The integral transform (1.11) defined on the space of U(n, K)-invariant functions 
on B„(K) is called the spherical transform. We observe that the spherical trans- 
forms are special cases of the index hypergeometric transform. The inversion 
formula (1.2) gives the following density for the Plancherel measure 

r{b + is)r{c + is) 

"(^^ = — fm — 



(1.13) 



§2. Correspondence of some operators 

2.1. Operator of multiplication by x. 

Theorem 2.1. Assume that a function f defined on M"*" is continuous and 
satisfies the condition 

f{x) =o{x-''-^-'); x^+oo. (2.1) 

Then 

ff{x)]b,c = P[mkc, (2.2) 
where the difference operator Pg is given by 

{b-is){c-is) . , . ss , {b + is){c + is) 

^^(^^ = {-2is){l-2is) (^(^ + ^) - + (2^a)(l + 2^.) ^^^^ " ^) " ^(«))- 

Proof. By condition (2.1), there exists a holomorphic continuation of the 
function / to the strip 

I Ims| < 1 + £. 

The identity (2.2) is equivalent to the following identity for hypergeometric 
functions 

X 2F\{b + is,b — is; b + c; —x) = 

(6 — is){c — is) 



{-2is){l - 2is) 
(6 — is) (c — is) (b + is) (c + is) 



2Fi{b — 1 + is,b + 1 — is;b + c; —x) — 

2-Fi(6 + is,b — is; b + c; —x) + 



{-2is){l - 2is) {2is){l + 2is) 

(b + is)(c + is) „ 
+ ' 2Fr{b+l + is,b-l-is;b + c;-x). 

[2is)(l + 2is) 

The last identity is equivalent to the following relation between associated hy- 
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pergeometric functions 
- y 2Fi{p,q;r;y) = 



q{r - p) 



{q-p){l + q-p) 
q{r-p) p{r-q) 



iq-p)il + q-p) {p-q){l+p-q) 
p{r - q) 



2Fi{p- l,q+l:r:y) - 

2Fi{p,q;r;y) + 



+ 



{p - q){l + p - q) 



2Fi{p+l,q-l;r;y). (2.3) 



The last identity is absent in standard tables but it can easily be checked by 
equating of the coefficients of the Taylor expansion in y. Indeed, 



2Fi{p-l,q+l;r;y)- 2Fi(p, q;r;y) = ^ 



ip)k{q)k fip-l)iq + k) 



(r)fefc! V {p + k-l)q 



jp- l)fc(g + l)fc _ {p)k{q)k 

{r)kk\ (r)fcfc! 

{p)k{q)k -k{^ + q-p) 



E 



(r)fcfc! q{p + k-l) 



Converting 2^1 (p +1,9— 1; ''S 2/) — 2Fi{p, q; r; y), we reduce the right part of 
(2.3) to the form 



E 



{P)k{q)k 

{r)kk\ \{q ~ p){p + k - I) {p-q){q + k-l) 



-k{r — p) 



'k{r — q) 



The last expression equals the left part of (2.3) 

2.2. Differentiation. Theorem 2.2. Let f , /' be continuous and satisfy- 
ing the decreasing conditions (2.2). Then 



[x{x + l)—f]t,c^H[f]b,c 

ax 



(2.4) 



where the difference operator H is given by 



T-r , \ (b — is)(b + 1 — is)(c — is) , , , 

= (%(i-2;l) (g^- + ^) - + 

(5 + is)(6 + 1 + is)(c + is) , , ., , , ^ 

+ (^2..)(i+2;l) - ^) - -^(^^ - + 



*For any triple of hypergeometric functions 2F\(p,q;r;y), 2F\(j> + k\,q + 1\;t + ra\]y), 
2F\{p + k2,q + l2\r + m2\y) with integer kj , Ij , nij there exists a linear relation with coeffi- 
cient, which are polynomial in ys. 
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Proof. 

d 



dx — 



ixix + I) — fix)] 2Fi(b + is,b- is; b + c; -x)x''+''-\l + xf-'' 
I dx } 

+ xf-''+^ 2Fi{b + is,b- is; b + c; -x)'jdx = 

f{x)lx{x + 1)— 2-^1(6 + is,b~ is; b + c; —x) + 
^ dx 

+ ((26 + l)x +{b + c)) 2^1(6 + is,b- is; b + c; -x)^x''+'''\l + xf'^dx. 
Now the statement is reduced to the identity 

- ^x{x + l)-^ + (25+ l)a;+ (6 + c)| 2Fi{b + is,b - is;b + c;-x) = 

(b — is)(b + I — is)(c — is) ^ 

- (^2^s)ilJ^s) " ^ + ^ + ^ " ^ + -) " 

nb-is){b + l-is){c-is) {b + is){b + 1 + is){c + is) \ 

~ \ {-2is){l-2is) ^ (+2is){l + 2is) " 7 ^ 

X 2Fi{b + is,b — is;b + c; —x) + 

{b + is){b+l + is){c + is) ^ , . 

+ (+2..)(l + 2;;) ' 2P^ib+l + ^s,b-l-^s;b + c;-x). 

This is an relation between associated hypergeometric functions 

- |a;(a:: + 1)-^ + {p + q + l)x + 2Fi{p,q;r;-x) = 

= (x^ + x)^ 2Fi{p+l,q + l;r + l;-x)-{{p + q+l)x + r) 2Fi{p, q;r; ~x) = 
q{q+ l){r -p) 



{1 + q - p){p - q) 

p{p+l)(r-q) 



2Fi{p - l,q+ l;r; -x) - 2Fi{p, q; r; -x)) + 

( 2Fi{p+l,q- l;r;-x)- 2Fi(p, q;r; -x)) 



[1 + p - q){q - p) 

and can easily be checked by equating of the Taylor coefficients at x''. 
§3. Berezin kernels 

The space on the ball B„(IK) (the ball B„(K) is the symmetric space 
U(l, n; K)/U(l, K) x U(n,K)) admits a natural deformation that will be de- 
scribed in this section (for more details see for rank 1 symmetric spaces and 
[ p6| for symmetric spaces of arbitrary rank). The main purpose of the present 
work is an imitation of this deformation on a level of special functions. 

3.1. Positive definite kernels. Recall the definition of a positive definite 
kernel (for detailed discussion and references see |26|). Let X be a set. A 
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function L{x, y) on X x X is called a positive definite kernel if L{y, x) = L{x, y) 
and for all xi, . . . ,Xk G X the matrix 

/L{xi,Xi) ... L{xi,Xk)\ 
det : •.. : 

\L{xk,xi) ... L{xk,Xk)J 

is positive semidefinite. 

Let i be a positive definite kernel on a set X. Then there exists a Hilbert 
space H = H[L] and a system of vectors Vx G H, where x ranges X, such that 

1. the linear span of the vectors Vx is dense in H[L] 

2- {vx,Vy) = L{y,x). 

For each vector h e H[L] we assign the function fh{x) on X by 

fh{x) = {h,Vx) 

Thus the space H is realized as some space of functions on X. We denote this 
function space by H°[L]. Obviously, the vectors Va € H[L] correspond to the 
functions 

V>a{x) = L{x,a) 

and for each / e H° [L] we have the following equality (the reproducing property) 

{f,'Pa}Ho[L]=f{a). (3.1) 

This identity gives some (constructive, but not completely) description of the 
scalar product in H° [L] . It is said also that L is a reproducing kernel of the 

space H°[L]. 

3.2. Berezin kernels. The Berezin kernel Lg{z,u) on B„(K) is defined by 

Lg{z,u) = \l-{z,u)\-''. (3.2) 

In the cases IK = M,C the kernel Lg{z,u) is positive definite for all 9^0, and 
in the case K = H the condition of positive definiteness is 9 > 2. The group 
\]{l,n;K) acts in H°{Lg) by the unitary operators 

(c d) f^'^ = ^(('^ + ^'^^"'^^ + "^^^ I" + "I"'- 

3.3. Another version of the definition. Consider the kernel on B„(K) 
given by 

\l-{z, u)\o • 

Consider the spaces H{Me), H°{Mg) associated with this kernel. The group 
\]{l,n;K) acts in the space H°[Mg] by substitutions by the formula 

^^(c j)/W=/((« + ^c)-i(^ + '^rf))- (3-3) 
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It is easily shown that the constructions of 3.2 and 3.3 are equivalent. The 
canonical unitary U(l, n; K) -intertwining operator H°[Lg] H°[A'Ig] is the op- 
erator of the multiplication by the function (1 — (z, z))^/^. 

3.4. More material description of the space H°{Lg). The cases K = 
M, C, H are similar and we will consider the case IK = M as a basic example. Let 
us give a more constructive description of the space H°{Lg) in this case (with 
an additional restriction > n). 

For this aim we consider the space Vg of holomorphic functions in the ball 
B„(C), we equip this space with the scalar product 

m 



(f^9)ve = . / /(z).g(z)(l- (z,z)r-"-Mz. 



.(C) 



The universal covering group of the group U(l,7i) — U(l,7i;C) acts in the 
space Vg by the unitary operators 

Ug f{z) = /((a + zc)-i(6 + zd)) (a + zc)-' . 

Remark. For an integer 6 the family of the operators Ug defines a linear 
representation of the group U(l,7i). If is not integer, then 

(a + zc)-' = (1 + zca-i)-^e-'^('"'^+2-'='). 

It can easily be shown, that (ca~^, ca~^) < 1, hence {1 + zca-^)-' is canonically 
defined for < 1. Thus the operator d) defined up to a factor 

g27rfci6» jjence for a noninteger 9 we obtain a projective representation of the 
group \J{l,n) or (this is one and the same) a linear representation of its universal 
covering group. 

A simple calculation (with the Dirichlet integral, see, for instance, 1.8) 
shows that the system of functions z'[^ . . . z^" forms an orthogonal basis in Vg 
and 

lk^..-MP = 7|r"^. (3.4) 



Consider the function 

(^)fci+...fc„ai' ■ ■■an" fci k 



t \ /l - l'7^fel+...fc„"l^ • • • "n" fci fe„ /o r-\ 

^a(z) (l-_^Zjaj) = ^ ki\...kj ^1 • (2-^) 

k\ ,kn 



By equations (3.4)-(3.5), for any function f ^ Vg 

{f,Va)ve = /(a)- 

Thus the space Vg is defined by the positive definite kernel 

K{z,u)={l-{z, u))'". 
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A function holomorphic in the ball B„(C) is uniquely determined by its restric- 
tion to B„(R), hence we can consider the space Vg as a space of functions on 
B„(M). This is strictly the space defined by the Berezin kernel Lg on B„(K). 

Emphasize that in the space H°[Lg\ on B„(K) we have an action of the group 
U(l, n), which is larger than 0(1, for more details see [ p6[ . 

3.5. Plancherel formula. Emphasize, that the group U(l,n;IK) acts in 
the space L^(B„(K)) and in all spaces H°{Mq) by the same formula (1.7), (3.3). 
Consider the following operator, which coincides with operator (1.9) up to a 
functional factor 

1 /■ \\ - iz h\\-{n+l)r/2+l+is 



T{h+c)\T{e-b + ls)\^ J ■'^ ^|l_(2,2)|(n+l)^/4+l/2+z«/2 

B„(K) 

(3.6) 

As was shown by van Dijk and Hille for 6* > 6 the operator Re is a unitary 
operator 

H°{Me) ^ l2(S'™-i X M+,r(s)d/ids), 
where dh is a Lebesgue measure on the sphere and 



t{s) ^ 



T{e)T{e-b + c) 



r(6i - 5 + is)T{b + is)V{c + is)\^ 



T{2is) 



(3.7) 



Remark. If we remove the expression \V'^{6~b + is)\'^ from the denominator 
in (3.6), then in (3.7) the same expression will relocated from the numerator to 
the denominator. By some reasons, the normalization (3.6) is more convenient. 

Below in §5 we obtain Theorem 5.3 imitating the Plancherel formula (3.7) 
on the level of the index hypergeometric transform, the van Dijk-Hille theorem 
is a special case of Theorem 5.3. 

3.6. Radial parts of Berezin kernels. Again let us for definiteness 
consider the case K = R. Consider the space Vg of holomorphic functions 
defined in 3.4, and its subspace V^^^^^ {the radial Berezin space) consisting of 
0(n)-invariant functions. 

Obviously, the elements of the space V^''^"'' have the form 



g{zl + ... + zl) = Y, Cp{4 + ■■■ + zlf. 



Hence we can consider V^'"*''"'' as a space of functions depending on one variable 

u = zl + --- + zl 

lying in the disk |m| < 1. 

^ For the complex ball B„(C), the analogous overgroup is U(l,n) X U(l,n). For the 
quaternionic ball the overgroup is U(2, 2n) 
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Lemma 3.1. Vectors form an orthogonal basis in the space y°(") and 

"I" "I - (^/2),(^/2 + l/2)/ 

Proof. Obviously, the vectors are pairwise orthogonal, hence it is suffi- 
cient to calculate By orthogonality (3.4), we obtain 



P||2 = ||/'r2i...-Ur2VI|2 = 'ST ( ?! 11^2'=! ^2fe„ ||2 



\Kf = \\iz^+---+zirf= E 

fel^O,...,fen^O,X; kj=p 

E 



p\ Y2ki\...2kJ 



ki\...kn\J {e)2p 



It is sufficient to evaluate 

^ (fcl!...fcn!)2 



V '^k.\...2kj 



fei^O,...,fe„^0,X; kj=p 

Denote a summand of this sum by A/^^ ... fe^. Consider the generating function 
h{yi,...yn) := J2 ^ku-.knVi' ■ ■ -vt = 

fci>0,...,/c„SsO 

m=l fc=0 m=l 

Expression (3.9) is the coefficient at in the Taylor expansion of 

%,...,y) = (l-42/)-"/2 

and this gives the required statement. 

Spaces that imitate V^^^^ arc defined below in 5.1. 

3.7. 0rsted problem. As before, let b,c be defined by (1.12). For > b 
the representation U(l,n;K) in H°[Lg] is equivalent to the representation in 

L^(B„(K)). There arises the following question: 

Is it possible to write explicitly a unitary intertwining operator 

S: L^{B^{K)) ^ H°[Le] ? 

It is easy to obtain a general form of this operator. The operator 5 is a 
product of three operators 

S = const • Ra^MA, 
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where A is given by (1.9), Re is defined by (3.6), and 
M : L 



r{b + is)r{c + is) 2^^ f^u. ^0-b + is)T{b + is)r{c + is) 



T{2is) J V r(2is) 

is the operator of division by a function tp{s) satisfying the condition 

\i>is)\ = \r{a-b + is)\. 
Lemma 3.2. The operator S is given by 

where dm is the invariant m,easure on the ball and 



r{b + is)Tic + is) 



2 



A(a;) = const • / 'ip{s) . 2Fi{b + is,b — is;b + c;—x) ds. 

Jo ^ (2is) 

(3.10) 

Proof. Consider the function 1 from Vg. Its preimage with respect to the 
operator S can be easily cvahiatcd, it is given by (3.10), where x = \z\^ j (1— |zp). 
By the reproducing property, for any (7 e Vg we have ((7, 1) = (?(0). Since the 
operator S is unitary, any function f & L'^ satisfies 



= j A(-l^)rfm(^). 

iB„(K) \1-\ZY)^ 



Now the kernel of the operator 5 can be reconstructed by invariance arguments. 

Where arises a question, is it possible to find i/'(x) such that the integral 
(3.10) can be explicitly evaluated. For several years I tried to solve this problem 
and now I think that this is impossible. It seems that the simplest variant 

tp{s) = T{e-b-is) 

is best. 

In §6 I am trying to understand, is it natural to consider the integral (3.10) 
obtained in this way as a 'new' special function. 

I am inclined to think that the answer is affirmative by the following 'meta- 
physical' reason. To be definite, assume IK = M. Then the natural symmetry 
group U(l, n) of the space Ve is larger than the symmetry group 0(1, n) of the 
space _L^(B„(]R)). If we identify with Ve, then we force the group U(l,n) 
to act in L^(B„(M)). It is natural to think that enlarging of a symmetry group 
must imply a nontrivial analysis on the level of special functions. 

There is also another fact that seems pleasant for mc. For 3 series of semisim- 
ple groups 0(n,C), Sp(n, C), U(p, q) the corresponding A-function can be ex- 
pressed as a determinant consisting of A-functions (3.10) related to rank 1 sym- 
metric spaces. 



2 
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§4. Hahn polynomials: preliminaries. 



4.1. Definition. a,b,c > 0. The continuous dual Hahn polynomials (see 
[0, ill) are given by 



S'„(s^; a, 6, c) = (a + 6)„(a + c)„ 3F2 



-n, a + is, a — is 
a + 6, a + c 



Obviously, the expression ^^(s^; a, 6, c) doesn't change if we if we transpose the 
parameters b,c. The Kummer formula (see pj. Corollary 3.3.5.) 



3F2 



a, (3, J 
S, e 



T{e)T{S + e-a- 

r(e - a)r((5 + e - /3 - 7) 



a,S - (3,5 - J 
6,6 + e-p--/ ' 



implies that 5,1 (s^; a, 6, c) is invariant with respect to arbitrary permutations of 
a, 6, c. 

4.2. Orthogonality relations. The Hahn polynomials form an orthogonal 
basis in on half-line with respect to the weight 

r(a + is)r(6 + is)r(c + is) 



r(2is) 



and 



r(a + is)r(6 + is)r(c + is) 



r(2is) 



S'„(s^; a, b, c)Sm{s'^;a, b, c) dx = 
~ r(a + b + n)T{a + c + n)T{b + c + n) nlSm.n- 
4.3. Difference equations. Consider the difference operator 



£2/(s) = B(s)y(s + i) - (S(s) + D{s))yis) + D{s)yis - z), (4.1) 



where 



B{s) 
Dis) 



{a — is){b — is)(c — is) 
(-2is)(l - 2is) ' 
(a + is){b + is){c + is) 



(-|-2is)(l + 2is) 

The Hahn polynomials are the eigenfunctions of this operator 
£S'„(s^; a, 6, c) — n ■ Sn{s'^\ a, 6, c). 



(4.2) 



4.4. Index hypergeometric transform and Hahn polynomials. 
Lemma 4.1. The image of the function (1 + x)^"'^^ under the transform 



Jh.c is 



r(a — is)r(a + is) 
r{a + b)r{a + c) 
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Proof. The statement is reduced to the table integral [Q, 7.51.10. 
Lemma 4.2. The image of the function 



with respect to the transform Jh c is 
r(a — is)T{a + is) 



(l + x) 



-a — b 



r{a + b)r{a + c) 



3F2 



n, a + IS, a — IS ^ 
a + b, a + c ' 



Sn(s ■,a,h,c). 



T{a + b + n)r{a + c + n) 
Proof. We must evaluate the image of the function 

(1 + x)-^-' = ( i^^) " (1 + x)-'^-' = 



x + lj' ' \ 



C^(-l)"~'=(l + 0;)-"-'=-"+'=. 



fc=0 

But the images of the functions (1 + ^.^-a-c-n+fc -^^ere evaluated above. 
§5. Nonstandard Plancherel formulas 

5.1. Spaces i?^ ^. Fix positive numbers a, 6, c satisfying the conditions 

a > 6, a > c, 2a > 1. (5.1) 

Consider the space of holomorphic functions in the disk < 1 satisfying 
the condition 

^^\f{z)\\\^\z\^f^-^dz <oo, 

|z|<l 

where dz denotes the Lebesgue measure in the disk. 

Let us define the scalar product in VF" by the formula 

1 

\A<\ 

We denote by ^ the Hilbert space obtained in this way. 

Remark. The spaces ^ as linear spaces don't depend on 6, c and coincide 
with . But the scalar products in these spaces are different. 

Lemma 5.1. The functions z^ form an orthogonal basis in Wf^^ and 

■' ' " r(a + 6 + k)T{a + c + k)' ^^'^^ 



(/,g) - ^Y{2a~\) II 2Fi(a-6,a-c;2a-l;l-|z|^)dz. 
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Proof. Orthogonality of the functions z'' is obvious. Let us evaluate 



1 


7rr(2a 


-1) 


2 




r(2a- 


1)J 


1 




r(2a- 


1)J 


1 




r(2a- 


1)J 



|z|^*(l - 2^^i(a - 6, a - c; 2a - 1; 1 - \z\') dz 



\z\<l 

1 



^2fc+l 



(l-r2)2"-2 2Fi(a -6, a -c; 2a - 1;1- r2)dr 



(1 - w)'=w2a-2 ^p^(^^ - 6, a - c; 2a - 1; 



Then all is reduced to the table integral Q, 7.512.4. 

Remark. We observe that the radial Bcrczin spaces defined in 3.6 

are special cases of the spaces Wl^^: 



b = n/4-l/A; c = n/4 + l/4; a = 6* - n/4 + 1/4. 

5.2. Reproducing kernels of the spaces Wl^^. 
Lemma 5.2. The reproducing kernel of the space is 



_ r(a + &)r(a + c) 



a + b, a + c _ 



b + c 



; zu 



Proof. Obvious. Indeed, assume (pw{z) — K^J^z,w). Then for any holo- 
morphic function f{z) — X^fcLo '^kZ^ we have 

, i.r(a + + fc)r(a + c + fc) , I. I... 



k=0 



kir{b + c + k) 



^Ckw'' = f{w). 



fc=0 



Remark. Kernels K^^{z,u) are special cases of the kernels introduced by 
Gross, Richards H and also special cases of the kernels considered by Odzijewicz 



5.3. Plancherel formula. Let g G W§^. Consider the transform 



|r(a + is)|2r(fe + c) 



{l+x) 



— a—b , 



x + l 



X 2Fi{b + is,b - is:b + c; -x)x''+''^\l + x^-^dx 
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Theorem 5.3 The operator ^ is a unitary operator 

r{a + is)T{b + is)r{c + is) 2 



T{2is) 

Proof. The image of the function f{z) = under the transform J^^ is 

Sk{s^\a,b, c) 



T{a + b + k)T{a + c+ky 



(5.3) 



Thus the required statement follows from (5.2) and the orthogonality relations 
for the Hahn polynomials. 

5.4. Operator d/dz. Denote by the space of functions holomorphic 
in the disk |z| < 1 and smooth up to the boundary. Let a > 1. By Lemma 1.1, 
for / € W°° the function ^.f is holomorphic in the strip | Im,s| < 1 + e. 

Proposition 5.4. For each f e W°° and each a > I 

where L is the difference operator defined by (4.1). 

Proof. The image of the function z'' is given by (5.3), and the Hahn 
polynomials satisfy the difference equations (4.2). 

§6. A-function and its properties 

Motivation of the definition of the A-function introduced in this Section is 
contained in above in 3.7. 

6.1. Definition. Let a,b,c E M. Let x S K.+ . We define the A-function by 

00 







X 2Fi{b + is,b — is;b + c; ~x) ds. (6.1) 



Remark. The function AjJ^(x) admits a holomorphic continuation to the 
domain Re a, Re 6, Re c > 0. It will be more pleasant for us to formulate prop- 
erties of the A-function for real a, 6, c. 

I couldn't express the integral (6.1) in terms of standard special functions (see 
the list in ) by a finite number of algebraic operations (except some special 
values of 6, c, see below 6.6). I think that this is impossible. The integrand 
in (6.1) is slightly intricate, but the true reason of nontriviality of the integral 
(6.1) are unusual limits of the integration. Indeed, let us change the limits of 
integration (for real a,b,c) to 

-t-00 

(6.2) 
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This is equivalent to an evaluation of 

ReAtJx). 



(6.3) 



Then our integral becomes a Slater type integral (see ||3^], |T^) and the standard 
residue machinery allows to obtain some expansions of ReAg^(a;) into series. 
Nevertheless I couldn't find a nice final expression for ReAj^^(a;) using this 
approach. 

For some special values of b, c expression (6.3) can be evaluated explicitly 
(see 6.7) 

Substituting a; = to (6.3), we obtain the Barnes integral 



ReA^,,(0) = 



27rr(6 + c) 



+ 00 



r(a + is)T{b + is)r{b - is)T{c + is)r{c - is) 
T{2is)T{-2is) 



ds. 



Applying the standard Barnes method, (see ||l9|, |jl|) we represent this 

integral as a sum of two hypergeometric series 4^4(1) 



ReA^^,(0) = 



(-l)"r(a - 6 - n)r(26 + n)T{c -b- n)r{c + b + n) 



E 



n!r(26 + 2n)r(-2&-2n) 

(-l)"r(a - c - n)T{2c + n)r{b - c - n)T{b + c + n) 
n!r(2c + 2n)r(-2c-2n) ' 



We observe that A^^(O) is some kind of a Barnes integral over a nonclosed con- 
tour. Nevertheless the expression A^^(O) is 'better' than the 'indefinite Barnes 
integral' 

- r(a + ,,^ib + ^s)Tibj^ic + is)ric-^s)^^^ ^^^^^ 



27rr(6 + c) 



r(2is)r(-2is) 



since the point s = is a distinguished point for the second factor of the 
integrand. 

6.2. Imitation of 0rsted problem. Consider the space W§^. The 
operator J^_^ is a unitary operator from Wj^^ to the space 



r{a + is)T{b + is)r{c + is) 



T{2is) 



ds 



The multiplication operator 

Mg{s) =^T{a + is)g{s) 
is a unitary operator from the space (6.5) to the space 

T{b + is)r(c + is) 2 



(6.5) 



(6.6) 



T{2is) 



ds 
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and the inverse index transform {Jt.c) "'^ is a unitary operator from (6.6) to 

L^{R+,x''+'-\l + x)''-''dx). 
Thus we obtain the unitary map 

{Jb,cy^ o M o : Wf^^^ L'^{R+,x''+''-'^{l + xf-^dx). 
This map is an imitation of the operator 

Ve ^ L'(B„(K)) 

discussed above in 3.7. The function AJJ^ is the image of the function f{z) = 1 
under this map. 

6.3. Direct corollaries of definition of the A-function. The inversion 
formula for the index transform implies 



1 



KlJx) 2Fi{h + is,b~is]b + c]-x)x^+''-^{l + x)''-''dx = T{a + is) 



r(6 + c) Jo 

The Plancherel formula for the inverse index transform implies 



Kl,{x)Kl(x)x'+'-\l + xr-^dx 



T{a + is)T{b + is)T{c + is) 



ds = T{a + b)T{a + c)T{b + c). 



T{2is) 

(for a deduction of the last row see 0], 3.6). 

6.4. Differential— difference equations. The identity 

2F^{a,l3-r.-x) = {1 + xy-^'^ ^F,i^~an-f3;r,-x) 

(see §, vol. 1, 2.1(23)) implies 

AlJx) = {l + xr-''Al,{x). 

Differentiating the integral (6.1) in the parameter x by the formula 

2Fi{a,(3;r,x) = — 2^1(0 + 1, /3 + 1; 7 + 1; x), 
ax 7 



(6.7) 



we obtain 
d 



dx 



r(6 + c + i) 



r(a + is) 



T{b+l + is)T{c + is) 



T{2is) 



X 2Fi(b + 1 + is,b + I - is;b + c + 1; -x)ds = -Al+i ^{x 
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and finally 



dx 



Observe the following corollary from formulas (6.7), (6.8) 



(6.8) 



^hk,c+ii^) - (-l)'^+'(l + x)=+'-^-^-^(l + x)''+''-=^A?^,(x). (6.9) 
By the formula (see §,2.5.7) 

(2:^ + 7- 1) 2Fi{a,(3]r,x) = (7- 1) 2i^i(a,/3;7- 1;^;), 

we obtain 



1 



T{b + c-l) 



r(a + is) 



T{b + is)T{c- l + is) 



(c + is){c — is) X 



r(2is) 



X 2Fi(b + is,b — is;b + {c — 1); —x) ds. 
Representing (c + is)(c — is) in the form 

(c + is){c - is) = (c^ - a^) + (2a + l)(a + is) - (a + is)(a + 1 + is), 
we obtain 

(a;^+6 + c-l)A?.,(x)- 

= (c2 - a2)A^,,(:r) + (2a + 1)AZH^) - K^c'i^) (6-10) 

- xA?+,,,(x) = (c^ - a^ - - c + l)A?_,(x) + (2a + l)A^+i(a:) - A^+2(a:). 

(6.11) 

Certainly our list of difference equations is not complete. 
6.5. One integral. 
Proposition 6.1. for n = 0, 1, 2, . . . 



or 



2Re / Al^{x)A^,+''{x)x'+^-\l + x)'-'dx 
= (2a)2„r(a + c)T{b + c)T{a + b) 4F3 



n/2, -(n + l)/2, a + fe, a + c 
-n + 1,2a, 2a + 1 



;i • 

(6.12) 
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The expression 4F3 in our case is not a scries but a finite sum. The right 
part of (6.12) also can be represented in the form 

T{a + b)T{a + c)r{b + c)(2a)2„ - 

- nr(a + b + l)r(a + c + l)T{b + c)(2a + 2)2„-2 + 

[(n+l)/2] / _0k4-^\ 

+nT{b+c) V '-^^ ^^T{a+b+k)r{a+c+k)i2a+2k)2n-2k. 

k=2 



Proof. By the Plancherel formula, our integral equals 



T{b + is)r{c + is) 



r{a + is)T{b + is)r{c + is) 2 



r(2is) 



ds. 



(6.13) 



Re [r(a + is)T{a + is + n)\ 

1 f°° 

=- / \{a + is)n + {a - is) 

TI" Jo 

Lemma 6.2. 

(a + is)n + {a- is)n = 
=(2«)2n - n{2a + 2)2„-2(a + is){a - is) + 

[(n+l)/2] , , I. , 1\ 

+ n (2a + 2k)2n-2k{a + is)k{a - is)k = 



k=2 

[(«+l)/2] 

= (2a)2„ Yl 
3=0 



k\ 



(-ri/2)i(-(n + l)/2),(a - is)j{a + is)j 



j\{-n+l)j{2a)2j 

By Lemma 6.2, we write integral (6.13) in the form 

[(n+i)/2] /o^ ^ / , 1^/o^ 
(2a)2„ E + ^ 



fe=0 



j!(-n+l)j(2a)2j 



r(a + j + is)r(6 + zs)r(c + is) 



T{2is) 



ds 



[(n+i)/2] /ON / / , 1^/o^ 
= (2«)2n E ^~"r^!^7 r(6 + c)r(a + i + b)T{a + j + c). 



fe=0 



,7!(-n+l),(2a)2,- 



This implies the required result. 

6.6. Relations with the A-function. The function X{z, a) is given by 

X{z,a) := / z~*r{t + l)dt. 
Jo 

It will be more convenient for us to change notations and to consider the function 

/zoo 
z-*T{t + l)dt 
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(it is the same indefinite integral with another point of the origin). The function 
A is an element of a relatively exotic family of special functions (sometimes they 
are called the VoUerra type functions) . This family includes also the functions 



ti{z,p) 



tPzJdt 

r(t + i)' 



r(i + i)' 



tJ'{z,p,p) 
v{z,p) 



tPz^+Pdt 

r(p + i + i)' 

zJ+Pdt 

r(p + < + i)' 



Observe some superficial resemblance of these functions with (6.1). Indeed, 
these integrals are similar to the Barnes integrals, but we have 'incorrect' limits 
of the integration. The functions /z(z,p), iJ,{z,p, p) were appeared in the Volterra 
work (10) on the fractional derivatives with logarithmic terms (see also ||3^). 
The functions /i, v were intensively discussed in French and Belgian mathemat- 
ical journals in the first half of 1940ies (Humbert, Poll, Colombo, Parodi and 
others, for instance, see, |14 , [Q). The first mention on the A- function that I 
find is contained in the tables of McLachlan, Humbert, Poll [po| . 

A theory of the functions v, p is contained in 'Higher transcendent functions' 
[H, 18.3 (the book contains also a large bibliography on this subject). The 
integrals with the functions p, A can be find in the tables of Prudnikov, 
Brychkov, Marichev, see corresponding subsections in volumes 3-5. 



Let us show that the functions 



^l/2+fc,i(^)' 



can be expressed in terms of A*(z,6) (in particular, this is valid for the A- 
functions related with groups of the subseries 0(1, 2n + 1)). For instance, con- 
sider the function 



A?/2,i(:^) 
1 



7rr(3/2) 



r(a -|- is) 



T{l/2 + is)T{l + is) 



T{2is) 



2s\fx 



sin(2s arcsh \fx)ds 



V(a\- is)is{e^^' '^'■'^"'^ ^ - "^^^^ v^j 
(r(a + 1 + is) - aY{a + is)) x 



l + x) 



1 - xf''\ ds. 



We obtained a sum of 4 integrals that can be expressed by the function A* . 

Using formula (6.9), we obtain an expression of ^y2+k terms of 

the derivatives of A*(x). The case AJ5 2/2-1-; similar. 
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Observe that the derivatives of the function A* can be expressed algebraically 
by the same function A* (a:;) 

— A* (z, 6) - - / -z~*r(i + l)dt = 
dz z 

= / {T{t + 2)-r{t + l))z-*dt = ~X*{z,b+l) + -X*{z,b). 

z Jb ^ 

6.7. The cases of explicit evaluation for Re A|J^. Let fc, Z = 0, 1, 2, . . . . 

Let us show that the following functions can be evaluated explicitly 

ReA?/2+fe,i(x), ReAly^^iix). 

Let us calculate 

n + OC ^ 

Re A"y2 1 •= const / T{a + is)s'^- — -= sin(2s arcsh v^)rfs = 
— const / (r(a + 1 + is) — aT{a + is)) x 



X iiVx^ + 1 + x)^'' ~ iV^F+T - x)^''] ds. 
Using the formula (see, for instance, iQ, (2.4.1)) 

1 p+ica 

/ V{u + t)z^dt = z'^t 

we obtain 



const 



{x + v/x2 + l)2«((.T + v/2;2 + 1)2 - a) exp{-(2; + ^x'^ + l)''] 



- (v/^E2+T-.x)2«((v/^2TT-a;)2 - a) exp{-(Va?2+T - a;)2} 

To obtain an explicit expression for Re A"^2+/c ; (^) i can apply formula 
(6.9). 

Remark. Slight inconvenience of the formulas of this and the previous 
subsection disappears after the substitution 

x = sh^ t. 

6.8. Comments. At the end of §3, I emphasized that there is some arbi- 
trariness in the choice of the A-function. Some of properties of our list survive 
for arbitrary choice of the function ■0 (• (3.10)), i.e. for general integrals having 
the form 



r 



, ,^,^^u(s) ^ib + ^s)T{b~^s)^{c + ^s)T{c-^s) 



7rr(6 + c)y ' ' r(2is)r(-2is) 



X 2Fi{b + is,b — is;b + c; —x) ds, 
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where Im u(s) = 0. This is vaUd for integrals from Subsection 6.3, for differential- 
difference equations (6.7)-(6.9), and for Theorem 7.1, Proposition 7.2, Theorem 
8.1. 

It seems important, that the A-hmction has a collection of properties that 
are not corollaries of the motivation given in 3.7, these properties are valid only 
for '0 = 1 (or u = 0). This is the difference equation (6.10) and all material of 
6.5-6.7. 

It should be noted that the differential-difference equations obtained in 6.4 
and the expressions by the A-function are valid also for the indefinite Slater in- 
tegral (6.4). All other properties of the A-function obtained above don't survive 

in this generality. 

§7. Birefiected bases and generalized translate hyper- 
group 

7.1. Birefiected bases. In 5.3 we defined the imitary operator 

By 

En{x) = En{x;a;b,c) 
we denote the image of the function z" under the transform Q. 

Theorem 7.1. a) The system of the functions S„ forms an orthogonal basis 

in L^{R+,x'>+''-\l + x)''-''dx). 

b) The functions S„ can be expressed in terms of the A-function by 



En{x;a,b, c) 



{b + c)n ^ {^iy{-n), b-c+i b+c+,-1 .g 

i-l (n -^ r^ (h d'T-J 



T{a + c)T{a + b + n) j^^j\ia + c)j{b + c)j dxi 

Proof. The statement a) is obvious and we must prove only b). Recall 
that the image of the function 2;" under the operator J^^ is a Hahn polynomial. 
Thus it is sufficient to evaluate the integral 



1 



T{a + b + n)T{a + c + n)T{b + c) J ' ' T{2is) 





00 

/r(. 



+ is) 



T{b + is)V{c + is) 



2 



X 



X 2Fi{b + is,b — is;b + c;—x)Sn{x •,a,b,c)ds. 

By the symmetry of the Hahn polynomials in a, b, c, we represent the integral 
in the form 



1 



r(a -I- is) 



r{b + is)T{c + is) 



T{a + b + n)T{a + c + n)r{b + c) J ' ' r(2js) 

2, c + is, 
a + c, b + c 



2 



X 



X 2Fi{b+is,b-is;b+c;-x){a+c)n{b+c)„ 3F2 ( "'I'Vt.^t' f ^ ;1 ) ds. 
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Expanding 3^2 into the sum, we obtain 
{b + c)„ 



Y.T, 



T{a + b + n)T{a + c)T{b + c) j-'^ j\{a + c)j{b + c)j 



X / r(a + is) 



T{b + is)r{c + j + is) . 
r(2is) I 



+ is, 6 — is; 6 + c; — x) ds. 



Applying the formula (|], vol.1, 2.8(22)) 

dy 

we obtain 



2^^i(a,/3;M;2/) = 2^1(0, + fc; y). 



(6 + c)r 



(-n)j- 



r(a + 5)r(a + c) j!(a + c),{b + c), r(6 + c + j) 



(-l)Ja; 



iT,l-6-c_^^6+c+i-l 



X / T{a + is) 



T{b + is)Tic + j + is) 



T{2is) 



2F1 {b + is, 6 — is; b+ {c + j); —x) ds. 



This gives the required expression. 

7.2. Generalized translate operator. The generalized translate hyper- 
group is an essential element of the theory of the index hypergeometric trans- 
form, (see 0, 0). We only will slightly touch on this subject. 

Consider the set J C M+ x x M+ consisting of all points (x, y, z) such 
that three numbers arcsh y/x, arcsh y/y, arcsh satisfy the triangle inequality. 
Consider the function K{x^ y, z) on M-|_ x IR+ x that is identical zero outside 
the set J and is equal to 



K{x,y,z) 



2-4fcr(6 + c) {{I + x){\ ^ y){\ + z)) 



c-l 



r(5 + c-l/2) {xyz)'>+'^-^ 
X (1 _ 52^6+^-1/2 2^^(25-1, 2c-l;6 + c-l/2;(l-B)/2) 

on the set J, where B denotes the expression 

^ ^ x + y + z + 2 

2{l + x){l + y){l + z)- 

The generalized translate operator L^(M+, + x)^~^) is given by 

nOO 

TJ{y) = / K(x, y, z)f{x)x'+'-\l + xf-^dx. 



It is known that 

Jb,cTz[Jb,cy^g{s) = 2Fi{b + is,b - is\b + c; -z) f [s) . 



(7.1) 
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Proposition 7.1. 



Kl,{x)Kl^{x)K{x, y, z) x'^+^'\l + xf-''y'+''-\l + yf-^dx dy 



= r(a + b)T{a + c)r(fe + c)(i + 

Proof. The expression in the left side is the scalar product A^^ Tz^.^ ^ in 
L2(M+, + xf-"). By the Plancherel formula (7.1), this equals 

T{h^is)T{c + is) 



T{2is) 



r(a + is)r(a + is) 2-F1 {b + is,b — is; b + c; —z) ds. 



The last expression is the inverse index transform of |r(a + is)p. But it was 
evaluated in Lemma 4.1. 

7.3. Generalized translate operator and bireflected bases. It seems 
that the generalized translate operator has a simple explicit matrix in the bire- 
flected basis. 

Author can evaluate only the first row. 

Proposition 7.3. 

TzKl^ = TzEoix; a, b, c) = (1 + z)"^-" ^ — — S„(x; a, b, c). 

Proof. The index transform of T^A^ ^ is 
r(a + is) 2F1 {b + is,b — is; b + c; —z) = 

= V{a + is){l + z)-^-'' 2Fi{b + is,c + is;b + c;^-) = 

z + 1 

= r(a + is){l + z)""-^ (1 ^) 2Fi{b + is,c + is; b + c; ^-). 

\ z + 1/ z + 1 

Using the following generating function for the continuous dual Hahn polyno- 
mials (see [0, p. 349), we obtain 



T{a + is)il + z)''-^J2 



Sk(x'^; a, b,c) f z 



k\{b + c)k \z + l 
This is equivalent to the required statement. 

§8. An application of A-function and of bireflected bases. 
Symmetric spaces U(p, g)/U(p) x U(g) 

8.1. The space U(p, q)/\J{p) x U(g). Assume p < g. Denote by U(p, q) the 
pseudounitary group of the order {p, q), i.e. the group of all matrices g = ( ° ^ ) j 
satisfying the condition 

a b\ fl 0\ fa bV _ fl 
c d) [0 -l) [c d) ~ lo -1 



28 



Denote by q the space of all matrices z over C having the size p x q 
with the norm satisfying |jz|| < 1. The group U(p, q) acts on Bp_g by the linear 
fractional transforms 

z z'^i := (a + zcy^{b + zd). 
The stabilizer K of the point consists of matrices having the form 



a 
d 



a e U(p),de U(g). 



The space Bp^^ is a homogeneous symmetric space, 

Bp^, = U(p,<7)/U(p)xU((?). 

The U(p, (7)-invariant measure on Bp ^ has the density det(l — zz*)^p^'^ with 
respect to the Lebesgue measure. The group U(p, q) acts in the space with 
respect to the invariant measure by the substitutions 



T(5)/(z)-/(z[^l) 



(8.1) 



8.2. Berezin spaces Hg(Bp^q). The Berezin space Vg(Bp^q) is the space 



of functions on q defined by the kernel 



K{z,u) 



dct(l - zz*y/'^ det(l 



uw 



|det(l-zu*)|« 



The group U(p, g) acts in Vg°(Bp,g) by formula (8.1). For sufficiently large Q 
the representation of U(p, (7) in Vg°(Bp ,j) is equivalent to the representation in 
L^(Bp_g) (see see also ^^). The 0rsted problem is to construct explicitly 
a unitary intertwining operator between these representations. 

8.3. A-function of the space U(p, g)/U(p)xU((7). Denote by Ai(z), 
the singular values of the matrix z. Define the new variables by 

x,(z) = A2(z)/(1-A2(z)). 

We define the A-function of the space U(p, q)/\J{p) x U(g) by 



Ap(z) 



A''(z) = det 



Si(a;i) 



^o{xp) \ 
Si(xp) 



•2) 



\^p_i(a;i) ... ^p-iixp)J 
where 

S,(x) = E,ix; e^{q+p- l)/2; (q - p + l)/2; {q - p + l)/2) 
are the first p elements of the bireflected basis. 
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Theorem 8.1. For g € JJ{p,q) by u we denote the image of the point 
€ Bp g under g. Then 



e/2 



A^(z)A«(z[fl) det(l - zz*)-P-'idz = dct(l - uu*) 



Corollary 8.2. Define the function L{z,u) on Bp^q x Bp^q by the formula 

where g„ €E U(p, q) is any element that transforms the point S Bp ^ to u. Then 
the operator 



Mf{z)= J L{z,u)f{u) dct{l ~ uu*)-P-'' du 

Bp g 

is a unitary \J{p, q) -intertwining operator 

Remark. The definition of the kernel L{z,u) can be reformulated in the 
following form. It is defined by the determinant (8.2), where the numbers Xj 
are the singular values of the matrix 

(1 - ZZ*)-1/2(1 - ZU*)il - UU*)-^/^. 

Remark. The similar statement is valid for the groups of series 0(n, C) 
Sp(2n,C). 
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